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FOREWORD 


This  annual  technical  report  presents  the  accomplishments 
during  the  period  of  1  June  1973  to  31  May  1974.  The  principal  investigator 
is  Dr.  D.  C.  Lai  and  co-principal  investigator  is  Dr.  T.  Kailath.  Since 
the  inception  of  this  project,  it  has  been  the  result  of  the  collaborative 
efforts  of  many  individuals.  In  particular,  Dr.  J.  E.  Anliker  has  spent 
endless  hours  on  this  project.  Most  of  the  staff  members  contributed 
in  the  writing  of  this  report. 


Biocybernetic  lactors  in  Human  Perception  and  Memory 

SUMMARY 

This  project  is  concerned  with  the  application  of  biocybernetic 
concepts  to  the  problem  of  expanding  human  memory.  In  particular,  the 
goal  of  this  research  is  enhancement  of  visual  imagery  as  far  as  possible 
in  the  direction  of  photographic  memory.  Scanpaths  of  the  eye  during 
inspection  of  visual  targets  are  treated  as  indicators  of  the  brain's 
strategy  for  the  intake  of  visual  information-  This  research  will  determine 
the  features  that  differentiate  scanpaths  associated  with  superior  imagery 
from  scanpaths  associated  with  inferior  imagery.  Similarly,  the 
tachistoscopic  exposure  of  the  visual  material  is  responsible  for  a  number 
of  adjustments  in  the  brain's  waking  rhythm;  the  electroencephalographic 
features  correlated  with  superior  imagery  will  be  differentiated  from  those 
correlated  wth  inferior  imagery.  A  computerized  biocybernetic  scheme 
will  be  imp1 emented  in  an  attempt  to  generate  image  enhancement  and  to 
train  the  individual  to  exert  greater  voluntary  control  over  his  own 
imagery.  In  this  technical  report,  we  give  an  account  of  our  accomplish¬ 
ments  made  in  this  reporting  period. 

We  made  significant  progress  in  eye-movement  measurement  and  tracking 
during  the  past  year.  New  techniques  for  eye-movement  prediction  and 
special  software  for  visual  scanpaths  study  have  beer  developed  and 
implemented.  A  real-time  scanpath  analysis  program  has  been  implemented. 
Progress  has  been  made  in  the  determination  of  features  that  differentiate 
scanpaths  associated  with  superior  imagery  from  those  associated  with 
inferior  imagei y  through  the  use  of  a  new  experiment  paradigm  and  various 


scanpath  analysis  schemes.  In  the  direction  towards  discovering  the 
temporal  cues  that  serve  as  the  keys  for  visual  memory  encoding  and 
retrieval,  we  made  a  study  of  the  visual  evoked  response  (VER)  contingent 
upon  the  phases  of  EEG  at  which  the  photic  stimuli  were  delivered.  We 
also  describe  an  KEG  signal  model  which  prescribes  the  stimulus-response 
relationship.  The  information  gained  in  this  model  will  elucidate  the 
behavior  of  the  alpha  frequency  and  phase,  and  thus  lead  to  a  more 
practical  brain  state  estimator  and  predictor  for  our  use  in  the  memory 
studies  of  this  project. 

During  this  reporting  period,  we  published  and  presented  six  papers 
at  various  international  and  national  scientific  conferences  in  addition 
to  papers  published  in  the  first  contract  year,  and  one  paper  has  been 
submitted  and  accepted  for  presentation  at  the  forthcoming  Conference 
on  Engineering  in  Medicine  and  Biology. 

This  research  project  has  significant  implications  ior  the  Department 
of  Defense  since  the  research  results  obtained  enhance  the  potential  for 
devising  new  and  unusual  techniques  that  will  permit  the  development  of 
stronger  imagery  in  men  with  ormal  memories.  Because  it  is  definitely 
an  asset  for  the  military  man  to  have  the  ability  to  absorb  information 
rapidly  and  to  retain  it  intact  for  long  period  of  time,  this  project 
will  attempt  to  intensify  the  post-stimulus  imagery  as  far  as  possible 
in  the  direction  of  photographic  memory--and  to  search  out  those  features 
of  successful  recall  strategies  so  as  to  enhance  the  individual's  power 
of  redintegration. 
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I .  INTRODUCTION 


The  objective  of  this  research  project  is  to  develop  biocybernetic 
concepts  and  techniques  useful  for  the  analysis  and  development  of  skills 
required  for  the  enhancement  of  the  concrete  images  of  the  "eidetic" 
type.  In  particular,  we  are  concentrating  on  the  problem  of  achieving 
biocybernetic  expansion  of  human  visual  memory  through  the  use  of  a 
closely-coupled  man-machine  system  which  perfoms  real-time  monitoring  and 
feedback  of  cues  that  serve  as  keys  to  memory  encoding  and  retrieval. 

This  approach  is  based  on  the  considerable  evidence  that  the  human 
nervous  system  depends  heavily  upon  spatial  and  temporal  cues  both  in 
the  encoding  and  decoding  of  memories,  especially  sensory  images.  One 
of  the  principal  limiting  factors  in  the  design  of  closely-coupled  man- 
machine  systems  is  our  limited  knowledge  of  human  memory  mechanisms. 

We  believe  that  closely-coupled  man-machine  systems  can  be  used  not  only 
for  the  measurement  and  prediction  of  human  mnemonic  performance,  but 
also  for  the  control  and  enhancement  of  human  mnemonic  skills. 

Our  scheme  is  to  develop  and  implement  techniques  for  real-time 
monitoring  and  prediction  of  central  nervous  activities  through  the 
analysis  of  eye  movements  and  KEG  signals.  We  have  made  significant  progress 
in  the  development  and  implementation  of  these  monitoring  techniques. 

The  information  obtained  in  the  monitoring  and  analysis  of  the  cortical 
activities  is  being  used  to  arrange  extraordinary  coincidences  between 
eye  pointing,  brain  states,  and  the  control  of  stimulus  patterns  using 
optimal  feedback  control  schemes.  The  visual  stimuli  are  presented  in 
various  ways  (projection  screen,  osc i 1 loscopic  displays,  or  in  complex 
tachistoscopic  batteries)  for  either  monopic  or  dichopic  viewing.  The 
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optical  and  elce trophysio) ogical  estimators  are  obtained  by  measuring 
the  eye  position  and  eye  movements  through  a  unified  system.  By  combining 
these  techniques  and  using  various  feedback  schemes  to  close  the  control 


loop  thereby  forming  a  closely  coupled  man-machine  system,  we  are  attempting 
to  achieve  greater  control  of  image  persistence  and  image  dissipation. 

Our  design  of  such  a  closely-coupled  man-machine  system  is  depicted 
in  Figure  1.1.  We  utilize  computer-based  models  in  our  system  config¬ 
uration  to  predict  (forward-time  analysis)  the  kind  of  stimuli  which  should 
be  used  in  order  to  produce  the  desired  future  responses.  The  KEG  signal 
and  the  eye-movement  measurement  are  shown  as  the  responses  which  are 
closely  controlled  since  the  models  are  designed  to  mimic  the  actual 
physiological  processes  in  regard  to  their  stimulus-response  relations. 

The  boxes  in  the  block  diagram  of  Figure  1.1  are  labeled  according  to 
their  functions.  All  except  the  interface  electronic  systems  are  to  be 
implemented  on  the  PDP-15  computer,  which  is  the  heart  of  the  implementation 
of  our  schemes.  Our  work  has  been  progressing  in  the  direction  of  achieving 
the  realization  of  this  closely-coupled  man-machine  system.  We  have 
designed  and  implemented  some  of  the  interface  electronic  systems  and 
signal  processing  algorithms.  Real-time  estimation  and  prediction  algorithms 
for  monitoring  brain  states  and  eye  pointing  are  being  designed.  The 
development  of  some  modeling  schemes  for  eye  movements  and  EKG  signals 
is  in  progress.  In  this  report,  we  present  our  accomplishments  in 
developing  and  implementing  those  concepts  and  schemes  which  are  essential 
for  the  realization  of  our  closely-coupled  man-machine  system  for  visual 
memory  enhancement. 

In  the  past  contract  year,  our  major  effort  was  devoted  to  eye- 


F.ye-movement  measurement 


Response 


Figure  1.1  Block  diagram  of  the 
closely-couplcd  man-machine  system 
for  visual  memory  tracking  and  training. 


movement  measurement  and  tracking  and  the  scanpath  study  for  the 
determination  of  spatial  cues  that  serve  as  keys  to  memory  encoding  and 
retrieval.  Progress  has  also  been  made  in  the  determination  of  the 
temporal  cues  which  are  thought  to  be  useful  for  memory  encoding  and  recall 
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A  phase  contingent  analysis  of  visual  evoked  responses  has  befn  made. 

This  study  shows  that  there  is  a  definite  excitability  cycle  correlated 
with  the  phase  of  the  alpha  rhythm  at  the  instant  of  photic  stimulation. 
This  will  be  described  in  Section  VII.  In  the  next  section,  we  describe 
new  calibration  procedures  and  distortion-correcting  schemes  for  our 
Biometrics  eye-movement  monitor.  This  improves  the  accuracy  of  our  eye- 
movement  data;  however,  we  have  exploited  the  full  capability  of  the 
Biometrics  eye-movement  monitor  system.  Wt.  foresee  that  more  accurate 
eye-movement  data  are  needed  in  our  future  work.  In  Section  III,  we 
present  our  progress  in  developing  new  techniques  for  eye-movement 
monitoring  and  tracking.  In  order  to  study  the  visual  scanpaths  associated 
w  1 1  h  superior  (eidetic)  imagery  and  those  associated  with  ordinary 
imagery  as  to  discover  the  spatial  features  that  are  more  memorable,  we 
devised  a  new  experiment  paradigm.  Its  methods  and  the  preliminary 
analysis  of  the  scanpath  data  are  presented  in  Section  IV.  Since  an 
effective  real-time  scanpath  analysis  is  essential  for  implementing  cyber¬ 
netic  models  for  the  prediction  and  feedback  of  ongoing  eye-pointing 
behavior,  we  have  developed  and  realized  such  a  real-time  scanpath  analyzer 
This  is  described  in  Section  V.  In  Section  VI,  we  present  an  EKG  signal 
model  that  relates  the  photic  stimuli  and  the  KEG  signal  and  some  new 
results.  The  understanding  of  thi£  stimulus-response  relationship  is 
essential  in  the  determination  of  the  temporal  cues  that  play  prominent 
roles  in  the  encoding  and  decoding  of  visual  memory.  As  mentioned 
previously,  we  describe  the  phase  contingent  averages  >f  visual  evoked 
responses  in  Section  VII.  Some  concluding  remarks  ard  future  work  are 
described  in  the  last  section. 
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1 1 •  CALIBRATION  AND  DISTORTION  CORRECTION 
OF  EYE -MOVEMENT  MEASUREMENT 


In  searching  for  the  spatial  and  temporal  cues  that  servo  as  keys  to 
encoding  and  recall  of  visual  memory,  we  need  an  accurate  technique  for 
the  measurement  of  eye  movement  and  eye  position.  At  present,  we  are 
using  the  Eye-Movement  Monitor  (Type  SG)  manufactured  by  Biometrics,  Inc. 
This  instrument  provides  a  simple  and  >  latively  easy-to-use  method  for 
measurement  of  horizontal  and  vertical  movements  of  the  eye.  In  the 
horizontal  direction,  the  method  is  based  on  the  measurement  of  the 
amount  of  infra-red  light  reflected  from  the  junction  of  the  iris  and  the 
sclera.  In  the  vertical  direction,  the  eyelid-sclera  junction  is  used. 

The  signal  produced  is  not  linear  and  is  susceptible  to  several  sources 
of  noise  which  affect  both  the  reproducibility  and  quality  cf  the 
measurements.  The  major  sources  of  difficulties  are: 

(1)  Movement  of  the  head  or  the  spectacles  on  which  the  sensors 
are  mounted  results  in  translation  of  the  X,Y  coordinate  space. 

(2)  C ross-tclk  between  X  and  Y  measurements;  i.o.,  horizontal  eye- 
movements  influence  "vertical"  output  voltages,  and  vice  versa. 

(3)  The  vertical  signal  suffers  from  an  "eyelid  hysteresis"  phenom¬ 
enon;  that  is,  the  eyelid  does  not  always  return  to  exactly 

the  same  position  near  the  corneo-scleral  junction  after  vertical 
movement  or  blinks. 

(4)  Nonlinearities  are  introduced  in  both  X  and  Y  signals  due  to 
changes  in  the  adjustment  of  the  apparatus  and  through 
differences  in  the  characteristics  of  the  eyes  of  the  subjects. 

Consequently,  we  are  confronted  with  the  problem  of  trying  to  compensate 
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for  the  noises  and  nonlinearities  in  the  e’ e  monitor.  The  response 
characteristics  of  the  eye-sensor  system  can  be  represented  by  curves 
similar  to  the  ones  shown  in  Figure  II. 1  where  only  the  l/orizontal  response 
curve  is  shown.  Given  the  output  (voltages  from  the  Bic|netrics)  we  are 
trying  to  find  ways  of  estimating  the  true  input  (eye  position). 

The  horizontal  component  Horizontal  signal 

of  actual  eye  position  w.  .»  voltage  from  Biometrics 


Figure  II. 1  Horizontal  response  curve 
for  eye-sensor  system. 

We  are  using  two  main  approaches  to  the  problem  of  compensating 
for  nonlinearities.  One  is  to  model  the  response  curve  via  piecewise 
linearization  techniques;  that  is,  we  attempt  to  linearize  the  nonlinear 
X  and  Y  voltages  to  match  the  linear  space  of  the  visual  stimulus.  The 
other  approach  is  to  translate  the  linear  features  of  the  visual 
stimulus  into  the  nonlinear  domain  of  the  eye  tracker  and  thus  achieve 
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a  closer  match  between  stimulus  and  response, 

In  the  case  of  piecewise  linearization,  we  let  (r  ,r  )  represent 
the  measured  X,Y  voltage  values  corresponding  to  some  stimulus  point 
then  the  most  genera..  foim  of  a  linear  mapping  between  r  ard 
s  is  of  the  form 


c  r  +  c  r  +  c 
11  1  12  2  13 


c  r  -f  c  r  +  c 
21  1  22  2  23 


The  coefficients  c^  .  are  defined  by  use  of  a  three  point  calibration 

array  which  empirically  determines  the  relationship  between  the  stimulus 

values  (s  ,s  )  and  the  response  values  (r  ,r  ). 

i  z  12 

Measured  value  Stimulus  value 


<rla’r2a> 

corresponds 

to 

<Sla'S2a 

<rib’r2b> 

corresponds 

to 

<Slb’S2b 

(rlc','2c) 

corresponds 

to 

(Slc’S2c 

This  correspondence  allows  us  to  set  up  two  sets  of  three  simultaneous 
linear  equations,  the  first  of  which  is 

Sla  =  C 1 1 r 1 a  +  C12r2a  +  C13 

Slb  -  cllrlb  +  C12r2b  +  C13 

lc  Lliric  1  C 1 2 1 2c  4  C 13 

Solution  of  these  eqiations  yields  the  required  coefficients. 

In  practice,  one  such  linear  mapping  is  not  sufficient  to  correct 
for  distortions  introduced  by  the  Biometrics  unit.  The  problem,  in 
general,  is  to  find  the  minimum  number  of  stimulus  calibration  points 
(and  thus  the  number  of  mappings)  needed  to  reduce  the  nonlinear  distortion 


to  a  level  below  the  other  noise  sources  mentioned  above. 

A  program  has  been  written  which  computes  the  e  coefficients 


lor  the  mappings.  It  consists  of  two  subroutine  calls  named  MAPSET  and 
MAPCNV.  The  visual  field  is  divided  into  three  to  eight  pie-shaped 
sectors  around  a  pivot  point  in  the  center  of  the  visual  field.  A  set 
of  coefficients  is  computed  for  each  sector  by  MAPSET.  The  second 
routine,  MAPCNV,  determines  which  sector  applies  to  a  given  measured 
point  (r  ,r  )  and  computes  the  corresponding  stimulus  value  (s  ,si ). 

The  program  IBALL,  originally  developed  to  gather  statistical  data 
about  the  response  curve,  was  modified  to  incorporate  this  piecewise 
linearization  technique.  The  program  basically  presents  various  points 
distributed  over  the  visual  space  on  a  storage  sco|  e  to  a  subject  These 
points  are  repeated  a  point-at-a-t ime  in  a  random  sequence.  The  responses 
from  the  Biometrics  unit  are  recorded  by  the  computer.  The  statistical 
properties  of  the  responses  corresponding  to  each  of  the  various  points 
are  then  computed.  Figure  II. 2  shows  tK.e  results  of  one  such  experiment. 
The  circles  show  the  points  which  were  presented  to  the  subject.  In  this 
case  each  point  was  presented  ten  times  in  the  course  of  the  experiment. 

The  cluster  of  spots  about  these  circles  indicate  the  actual  measured 
responses.  The  crosses  represent  the  statistical  properties  of  the 
individual  clusters.  The  lengths  of  the  horizontal  and  vertical  bars 
are  proportional  to  the  standard  deviation  of  the  error  in  each  of  the 
corresponding  directions.  It  is  seen  that  the  responses  arc  distorted 
by  the  nonlinsarities  and  vertical  crosstalk.  As  a  demonstration  of  the 
capabilities  of  piecewise  linearization,  the  means  of  each  of  the  distorted 
clusters  were  used  to  form  the  response  matrix  for  MAPSET  while  the 
coordinates  of  the  actual  stimulus  points  determine  th?  stimulus  ar-ay. 

The  measurements  recorded  via  the  Biometrics  were  then  remapped  by  MAPCNV 
and  displayed  as  shown  in  Figure  II. 3.  Figures  II. 4  and  II. 5  show  the 
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It  is  seen  that  the  piecewise  linearization  method  is  limited  to 
the  correction  of  small  distortions.  The  amount  of  distortion  is,  in 
turn  affected  by  the  noise  in  the  environment.  The  variations  in  the 
response  curve  for  the  horizontal  response  have  been  measured  to  range 
from  .4  to  1.2  degrees  of  visual  angle  and  those  for  vertical  response 
were  about  twice  the  horizontal  range. 

Another  means  of  compensating  for  nonlinearities  in  the  eye  monitor 
is  to  translate  the  outline  of  the  stimulus  into  the  nonlinear  domain 
of  the  eye  monitor  by  having  the  subject  trace  a  cartoon  of  the  linear 
stimulus  relating  actual  eye  position  to  output  signal.  The  general 
approach  is  schematically  shown  in  Figure  II. 6  A  subject  is  shown  a 
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Figure  II. 6  Compensation  of  nonlinearity 
by  nonlinear  means. 

stimulus  (say,  a  square)  on  the  storage  scope.  His  scanpath  response 
is  then  recorded.  The  subject  is  also  required  to  track  a  spot  of  light 
that  moves  along  the  stimulus  cartoon  stored  in  the  SEER  program.  In 
this  way,  the  nonlinear  scanpath  can  be  compared  with  a  similarly  non¬ 
linear  version  of  the  stimulus  cartoon  and  thereby  achieve  a  closer 
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III.  EYE -MOVEMENT  MONITORING  AND  TRACKING 

1 .  Introduction 

The  tracking  of  eye  movement  is  an  essential  part  of  our  research 
project.  In  order  to  feedback  the  spatial  cues  in  encoding  and  decoding 
of  memories,  we  have  to  track  constantly  the  movement  of  the  eyes. 
Basically,  for  the  kind  of  eye  movements  we  are  interested  in,  we  can 
classify  them  into  two  modes;  viz.,  fixation  and  sarcades.  Thus,  for  the 
purpose  of  monitoring  and  tracking  we  need  to  (a)  estimate  and  predict 
the  duration  of  a  fixation  when  in  that  mode;  i.e.,  to  predict  when  the 
mode  will  change  from  a  fixation  to  a  saccade  and  (b)  estimate  and  predict 
the  duration;  the  direction,  and  the  length  of  the  saccade  for  the  saccadic 
movement  itself,  i.e.,  to  predict  when  and  where  the  saccade  will  end. 

The  two  parts  are  interdependent.  Ultimately,  we  will  build  a 
model  which  will  take  care  of  ooth.  However,  more  insight  can  be  obtained 
if  we  deal  with  the  two  separately  and  then  combine  them  into  one  pre¬ 
dictive  model.  At  present,  we  have  looked  into  part  (b) .  For  the  sake 
of  simplicity,  we  will  first  develop  a  deterministic  model  which  ignores 
the  problem  of  the  direction  and  the  stochastic  properties  of  the  saccade 
(assuming  it  to  be  a  straight  line)  and  gives  a  prediction  of  the  final 
position  after  looking  at  the  onset  of  the  saccade.  The  estimation  of 
the  length  and  duration  of  the  saccade  is  the  more  difficult  part  of  the 
problem.  Some  empirical  observations  have  been  made  and  on  the  basis  of 
these,  we  have  developed  a  predictive  procedure — based  on  the  velocity  of 
the  eye  movement.  This  will  be  described  in  detail. 

2.  The  Prediction  of  the  Location  of  Eye  Fixation 


By  using  the  eye-movement  velocity  information,  we  developed  and 


implemented  a  scheme  for  prediction  of  a  new  eye  position  at  the 


termination  of  a  saccadic  movement  about  half  way  before  that  new  position 
has  been  reached  by  the  eye.  The  scheme  is  based  on  the  observation  of 
eye-movement  velocity.  The  velocity  during  a  saccade  increases  to  a 
maximum  value  and  then  decreases  back  to  a  low  value.  It  has  been  observed 
that  the  velocity  curve  is  not  symmetrical  around  the  peak.  In  particular, 
a  '  trailing  off"  of  the  velocity  at  the  end  of  the  saccade  can  be  clearly 
observed  in  most  cases.  However,  it  has  been  seen  that  the  eye  travels 
half  the  saccadic  distance  at  the  peak  velocity  point,  thus  leading  to 
the  conclusion  that  the  areas  under  the  saccadic  velocity  curve  before 
and  after  the  peak  are  equal  in  spite  of  the  asymmetry  in  the  velocity 
curve.  Thus,  by  detecting  the  peaks  in  the  velocity  curve,  the  distance 
the  eye  has  moved  since  the  beginning  of  the  saccade  up  to  the  time  of 
this  peak  can  be  doubled  to  get  a  prediction  of  the  eye  position  at  the 
end  of  the  saccade  when  the  eye  has  arrived  at  a  new  fixation  point. 
Theretore,  when  a  saccade  is  only  half  completed,  a  prediction  of  the 
eye  position  at  its  end  can  be  made.  Because  of  the  asymmetry,  the  saccade 
duraction  cannot  simply  be  obtained  by  doubling  the  time  to  the  velocity 
peak.  However,  the  relationship  between  saccade  duration  and  length  has 
been  reported  in  earlier  studies  [l]  and  can  be  used  to  predict  the 
du rat  ion . 

\  block  diagram  of  the  scheme  is  shown  in  Figure  III.l.  The  X  and 
Y  eye-movement  voltage  outputs  of  the  Biometrics  Eye-Movement  Monitor 
are  filtered  by  analog  6-polc  Bessel  filters  with  cut-off  frequencies 
ranging  from  50  Hz.  to  200  Hz.  The  outputs  of  these  anti-aliasing 
filters  are  sampled  at  1000  Hz.  (1.0  ms)  or  500  Hz.  (2.0  ms)  by  the  A/D 
conversion  system  of  the  PDP-15.  This  digitized  eye  position  data  may 
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Figure  III.l  Block  diagram  of  prediction 
scheme  using  velocity  information. 


then  be  smoothed  by  various  digital  filters  implemented  as  subroutines. 

The  eye-position  data  points  are  scaled  according  to  a  calibration- 
determined  mapping  function.  Eye  "velocity"  is  then  calculated  from 
these  sealed  position  values  by  taking  the  difference  between  successive 
values : 

"velocity"  =  X(n)  -  X(n-l) 

where  X(n),  X(n-l)  are  two  successive  scaled  eye-position  values. 

This  result  is  divided  by  the  sampling  period,  T,  to  obtain  the  correct 
magnitude  of  the  actual  velocity. 

During  a  fixation,  this  velocity  fluctuates  about  zero  with  some 
variance,  which  is  determined  for  each  subject  in  each  experiment. 

A  threshold  detector  is  used  to  monitor  this  fluctuation.  When  the 
velocity  increases  monotonically  above  the  maximum  fluctuation  level, 
it  is  decided  that  a  saccade  has  begun.  At  this  time,  the  on-going 
means  of  the  X-Y  positions,  which  have  been  calculated  throughout  the 
fixations,  are  saved  for  later  analysis.  The  peak  detector  now  begins 
to  look  for  peaks  in  the  velocity  curve.  When  a  peak  is  detected,  the 
X  position  value  at  the  peak  time  is  used  in  the  equation  below  to 
predict  what  the  final  position  will  be  at  the  end  of  the  saccade: 

final  position  peak  initial  initial 

=  2X  ,  ~  X 

peak  initial 

where  is  the  mean  value  of  the  X  position  which  was  stored 

at  threshold  detection.  The  same  calculations  are  made  simultaneously 
to  predict  the  Y  position. 

As  an  example,  we  show  the  prediction  process  in  Figures  III. 2 
through  III. 6.  In  each  figure,  we  show  the  eye  movement  on  an  X-Y 
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plot,  the  horizon t ;i  1  (X)  velocity  vs.  time,  and  the  horizontal  (X) 


eye  position  vs.  time.  In  )  igure  III. a,  we  show  the  onset  of  the 
saccade.  The*  intermediate  eye-movement  processes  are  shown  in  Figuies 
III. 3  and  III. 4.  In  Figure  III. 5, the  velocity  reaches  its  peak  and 
the  prediction  ol'  the  fixation  point  is  made.  Finally,  in  Figure  III. 6 
we  show  that  the  eyes  arrive  at  the  predicted  fixation  point. 


X-Y 
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Figure  III. 2  Onset  of  saccadic  movement 


The  mot  hod  just  described  is  simple  and  useful;  however,  it 


restricts  us  to  making  predictions  only  when  the  eye  reaches  the  hallway 
point  of  *he  saceade.  It  is  desirable  to  make  a  prediction  of  final 
eye  position  at  a  mueh  earlier  time  of  the  saecade.  One  possible  means 
to  this  end  is  to  model  the  saeeadic  process  and  make  estimates  of  the 
parameters  of  the  model  based  on  information  contained  in  the  first 
severe l  points  of  the  saccade.  Yarbus  [l]  reports  that  the  velocity 
during  a  saccaue  follows  a  sinusoidal  curve.  We  eould  fit  a  sinusoid 
to  the  first  few  points  of  the  saeeade  and  thus  prediet  the  peak. 

However,  the  sinusoidal  function  is  infinitely  derivable  and  hence 
susceptible  to  noise.  We  have  decided  to  use  low  order  polynomials 
instead.  We  are  in  the  process  of  implementing  this  scheme.  In  addition, 
an  analysis  of  saccades  to  obtain  a  better  model  is  in  progress. 

It  has  been  hypothesized  that  the  eye  does  not  observe  anything 
during  the  saccadic  movement.  Thus,  before  the  saceade  is  made, 
the  decision  is  made  about  where  to  move  the  eye  and  with  what  velocity, 
lay  information  about  these  parameters  in  the  eye-movement  data  should 
th<'n  be  present  in  the  data  immediately  preceeding  the  saeeadie  movement. 
Thus,  by  analyzing  the  fixation  data  just  before  the  onset  of  a  saceade, 
we  can  determine  in  what  form  this  information  is  present  and  whether 
wt  can  use  it  to  predict  the  onset.  Essentially,  some  veloeity  and 
direction  analysis  must  be  performed  on  this  data. 

for  this  purpose,  the  onset  of  a  saccade  is  determined  by  the 
velocity  of  filtered  data.  When  this  velocity  crosses  a  certain  threshold, 
we  mark  the  end  of  the  saeeade.  Then  a  window  of  data,  consisting  of 
the  saccade  and  some  samples  before  and  after  the  saccade  (e.g.,  frun 
point  P  to  Pj)  are  eolleeted.  For  analysis,  all  saccades  of  one  length 
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arc  averaged  with  the  beginning  points  lined  up.  This  smooths  out  the 


noise  and  generates  a  "typical  saccade"  of  a  given  length  which  can  be 
used  to  design  a  model.  In  another  approach,  the  saccades  are  averaged 
with  their  peak  velocity  points  lined  up.  From  this  we  can  determine 
the  characteristics  of  the  asymmetry  about  the  peak  and  the  relationship 
between  the  saccade  length  and  the  peak  velocity. 

3.  L/iscussion 

Our  ultimate  aim  in  the  studying  of  the  characteristics  of  saccades 
and  fixations  is  to  study  the  overall  viewing  of  a  stationary  scene 
as  n  whole.  It  has  been  hypothesized  that  the  scanpaths  play  an  important 
role  in  the  study  of  visual  memory  and  we  have  referred  earlier  [2] 
to  some  theories  of  visual  learning  based  on  the  observation  of  repetitive 
strings  of  fixation  points  during  viewing  of  stationary,  two-dimensional 
patterns.  In  the  past,  however,  the  study  of  scanpaths  has  been  mostly 
qualitative,  with  determination  of  the  existence  of  repetitive  scanpaths 
being  based  solely  on  observation  of  the  eye-movement  plot  by  eyeballing. 
We  have  started  to  obtain  a  more  quantitative  procedure  so  that  the  mass 
of  data  can  be  reduced  to  a  point  that  one  can  introduce  some  degree 
of  objectivity  in  the  Jeterminat ion  and  characterization  of  the  visual 
scanpaths . 

One  approach  is  as  follows:  The  field  of  view  is  divided  into  a 
rectangular  NXN  grid.  Data  are  collected  for  many  viewings  of  a  given 
pattern.  This  is  separatdd  into  fixations  and  saccades,  each  saccade 
being  characterized 'by  a  starting  cell  and  an  ending  cell.  Now  for 
each  cell  which  is  a  starting  cell  at  least  once,  we  can  get  a  histogram 
of  the  ending  cells  which  will  determine  if  there  is  a  preferred  place 
to  which  the  eye  travels  from  a  certain  cell.  By  connecting  the 
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high  occurrence  saccades,  we  can  generate  a  scanpath. 

It  was  observed  from  this  approach  that  many  times  the  movement 
between  two  points  is  broken  by  intervening  fixations  on  the  same 
line.  Hence,  instead  of  the  ending  cell  histograms,  direction  (of  the 
saccade)  histogram  from  each  starting  cell  may  be  more  useful.  Work  on 
this  is  in  progress. 
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IV.  SCENE-DIFFERENCE  STUDY:  METHODS  AND  ANALYSIS 


1 .  The  Scenc-d if fercncc  Experiment 

Sensory  information  processing  systems  in  nature,  and  the  visual 
system  of  man  in  particular,  arc  extremely  senstive  to  differences  in 
the  properties  or  characteristics  of  certain  kinds  of  stimuli.  Classical 
studies  of  stereo  acuity  and  vernier  acuity,  for  example,  demonstrate 
how  well  two  types  of  visual  discrimination  can  be  made.  Therefore, 
as  a  goal  for  our  research  in  visual  memory,  we  decided  to  pursue  the 
idea  of  measuring  visual  memory  capacity  through  the  measurement  of 
ability  to  detect  differences  in  two  nearly  identical  scenes. 

The  experimental  procedure,  in  essence,  is  as  follows:  An  observer 
is  presented  with  a  visual  scene  or  pattern  ("the  inspection  figure”) 
and  is  told  to  look  at  it  carefully  and  remember  as  much  about  it  as 
possible.  Some  time  later,  another,  nearly  identical,  scene  is  presented 
("the  test  figure")  and  the  observer  is  asked  whether  or  not  the  scenes 
are  identical. 

By  running  such  an  experiment  we  wish  to  gain  additional  insight 
and  to  collect  data  on  some  of  these  questions: 

1.  What  can  be  said  about  similarities  or  differences  in  the 
scanpath  used  by  different  subjects  or  the  same  subject  in 
viewing  the  inspection  figure? 

2.  What  similarities  are  there  between  the  "inspection  scanpath" 
and  the  "test  scanpath"? 

3.  What  scanpath  strategy  is  used  by  different  subjects  in 
detecting  differences  between  scenes? 

4.  What  types  of  changes  to  the  scene  are  easily  detected  by  our 
observers,  and  what  changes  are  not? 
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5  What  differences  in  test  scores  or  scanpaths  may  be  iound  b\ 
subjects  with  eidetic  imagery? 

6.  EEG  correlates  to  the  scanpath  and  detection  strategies. 

Thus,  there  are  dual  uses  for  data  obtained  from  this  study: 
analysis  of  the  methods  used  for  detecting  scene  differences,  and  analysis 
of  repeated  scans  of  the  same  visual  material. 

The  stimulus  which  we  selected  for  our  intital  studies  is  the  still 
life  scene  shown  in  Figure  IV. 1.  In  designing  an  experiment  such  as 
this  there  is  a  wide  range  of  visual  stimulus  material  available, 
roughly  divided  into  the  following  three  categories. 

a.  Highly  detailed  scenes,  rich  in  colors  and  textures. 

b.  Impoverished  scenes,  such  as  black  and  white  line  drawings. 

c.  Degraded  scenes,  low  in  contrast  or  containing  much  visual 
noise. 

We  chose  our  stimulus  from  the  first  category.  This  allows  us  to  compare 
our  results  with  many  previous  studies  of  scanpath  measurements  of 
natural  scenes,  and  the  impoverished  and  degraded  images  required  lor 
the  second  and  third  categories  may  be  obtained  from  the  original  scene. 

One  difficulty  with  using  such  a  highly  detailed  scene,  however,  is 
that  it  becomes  very  difficult  to  quantitatively  measure  the  magnitude 
of  any  changes  made. 

2 .  Experimental  Method 

A  series  of  35  mm  slides  was  prepared  portraying  the  still  life 
as  shown  in  figure  IV. 1.  One  of  these  slides  was  designated  the  "standard’’ 
slide  and  the  remaining  slides  differed  from  the  standard  in  some  way, 
usually  through  an  addition  of,  deletion  of  or  change  ol  one  component 
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st.  Ul(i,  ircl  scene 


in  the  still  life  scene.  The  slides  were  projected  on  a  ground  glass 
screen  located  41  cm.  in  front  of  the  subject.  At  this  distance  the 
slide  covered  an  angular  field  of  30°  width.  Eye  movements  were 
recorded  using  a  Biometrics  SGH/V-2  monitor.  The  vertical  and  horizontal 
output  signals  were  digitized  using  an  analog-to-digital  converter  and 
then  written  on  a  digital  magnetic  tape.  A  computer  analysis  of  the 
digitized  data,  using  programs  and  algorithms  described  elsewhere  in 
this  report,  converted  the  input  data  into  a  sc  uence  of  fixation 
positions  and  fixation  start-stop  times.  Figure  IV. 2  is  a  record  of 
the  scanpath  derived  by  this  program  superimposed  on  a  sketch  oi  the 
original  scene. 

The  subjects  were  undergraduates  at  Stanford  and  each  had  20/20 
vision.  Eye  positions  were  calibrated  using  a  9-point  calibration 
array. 

An  experimental  session  consisted  of  the  presentation  of  25  pairs 
of  slides,  the  first  slide  of  each  pair  being  the  standard  slide.  The 
standard  was  presented  for  10  seconds,  and  then  the  test  slide  was 
presented  after  a  delay  of  one  second.  The  subject  was  instructed  to 
press  one  of  two  buttons,  one  button  corresponding  to  a  change  in  the 
scene,  and  the  other  corresponding  to  no  change.  The  slide  was  removed 
as  soon  as  either  button  was  pressed,  and  after  a  short  delay,  the 
standard  appeared  once  again.  The  sequencing  of  slides  and  running  of 
the  experiment  was  under  computer  control. 

3.  Preliminary  Analysis  of  Scanpath  Differences 

This  experiment  was  run  by  using  a  number  of  different  observers. 

Our  initial  attempt  in  analyzing  the  data  is  to  gain  insight  for 
characterizing  the  scanpaths.  Figures  IV. 3  through  IV. 6  are  representative 


of  the  results  of  this  preliminary  analysis.  Our  attempt  is  to  study 
all  possible  means  for  quantative  characterization  of  the  visual  scan- 
paths.  Since  a  scanpath  is  composed  of  a  sequence  of  fixations  and 
saccades,  we  try  to  scrutinize  various  properties  of  the  fixations  and 
saccades  in  this  preliminary  analysis.  In  Figure  IV. 3,  we  plot  the 
number  of  fixations  versus  elapsed  time.  The  elapsed  time  in  seconds 
from  the  presentation  of  the  slide  is  given  on  the  horizontal  axis.  The 
total  number  of  fixations  is  given  on  the  vertical  axis.  The  slope  of 
the  line  is  approximately  four  fixations  per  second,  which  means  that 
the  average  saccade  plus  fixation  time  is  250  milliseconds.  In  order 
to  ascertain  some  statistical  properties  of  fixations,  we  made  plots  of 
fixation-duration  histograms.  A  typical  histogram  is  shown  in  Figure  IV. 4. 
Fixation  times  were  sorted  into  50  millisecond  bins  and  the  resulting 
histogram  is  plotted  in  the  figure.  Mean  fixation  duration  time  for  the 
single  scan  shown  is  215  milliseconds.  In  Figure  IV. 5,  we  present  a 
plot  of  fixation  duration  versus  fixation  number.  This  graph  illustrates 
the  fixation  time  for  each  fixation  in  the  scan.  A  systematic  study  of 
saccadic  distances  has  been  made.  A  typical  result  is  shown  in  Figure  IV. 6 
The  distance  in  degrees  of  visual  angle  between  sequential  fixation  points 
is  plotted  on  this  graph.  A  small  angle  approximation  was  used  to  compute 

the  distance. 

4 .  Foveal  Fixation  Plots 

One  benefit  of  computerized  recording  and  analysis  of  scanpaths 
is  that  it  permits  transformations  of  the  data  which  can  lead  to  insights 
which  would  otherwise  be  lost.  A  graphic  illustration  of  our  ability 
to  perform  such  transformations  is  shown  in  the  foveal  fixation  plots 
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of  figure  IV. 7.  Using  the  first  25  fixation  points  shown  in  Figure  IV. 2, 
we  have  plotted  a  sketch  of  the  visual  stimulus  impressed  on  the  fovea 
for  each  fixation  point.  The  sketches  show  a  field  6°  in  diameter, 
a  region  which  includes  all  of  the  fovea  under  practically  any  definition. 
These  drawings,  then,  represent  a  time  sequence  of  snapshots  which  are 
the  visual  input  to  the  observer  in  the  experiment.  He  must  create 
a  unified  picture  of  the  object  or  objects  being  viewed  from  the  sequence 
of  picture  fragments  shown  in  the  sketches. 

Our  interest  in  this  type  of  fixation  point  representation  has 
three  sources.  First,  as  discussed  above,  it  is  an  excellent  illustration 
of  the  problem  of  mapping  scattered  pieces  of  the  visual  world  into  a 
unified  whole.  Next,  the  sketches  confirm  the  fact  that  the  great 
majority  of  "next"  fixation  points  are  not  present  in  the  "current" 
foveal  region.  Independent  verification  of  this  fact  is  contained  in 
figure  IV, 6,  which  shows  that  the  mean  number  of  degrees  per  saccade  is 
7.8,  greater  than  twice  the  radius  of  these  sketches.  Finally,  isolation 
of  the  foveal  fixations  gets  right  to  the  problem  of  understanding  why 
certain  regions  of  a  visual  scene  are  more  heavily  favored  for  fixations 
than  others.  Merely  saying  the  "information  content"  of  these 
regions  is  higher  than  that  of  their  neighbors  does  not  clarify  the 
issue.  We  would  like  to  be  able  to  analyze  this  information  content 
in  other  than  a  very  qualitative  way. 

5.  Seanpath  Similarity  Measures 

In  studying  the  scanpaths  created  by  different  observers  in  this 
experiment  it  is  very  desirable  to  have  some  measure  of  the  similarity 
between  two  scanpaths  or  seanpath  segments.  We  are  currently  exploring 
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several  functions  which  give  us  an  objective  measure  of  the  similarity 
between  two  fixation  points  and  the  similarity  between  two  saccades. 

We  have  used  the  saccade  similarity  function  to  derive  a  special  type 
of  saccade  density  map  showing  the  most  heavily  traveled  routes  in  the 
visual  field.  We  attempted  to  use  these  functions  to  detect  cyclic  or 
periodic  sections  in  scanpaths.  Works  pertinent  to  this  area  are  in 
progress. 
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V.  REAL-TIME  SCANPATH  ANALYSTS 


We  have  reported  previously  on  automatic  methods  for  the  off-line 
analysis  of  scanpaths.  We  shall  now  describe  a  real-time  method  for  the 
analysis  of  scanpaths.  An  effective  real-time  scanpath  analyzer  is  an 
essential  development  in  the  process  of  implementing  cybernetic  models 
for  the  prediction  and  control  of  ongoing  eye-pointing  behavior. 

This  computer  program  has  been  implemented  on  an  SEL-840  computer 
which  has  an  Evans  and  Sutherland  display  system.  Two  large  CRTs  are 
utilized:  (1)  a  monitor  scope  on  which  are  displayed  the  various  analyses 
that  are  of  interest  to  the  experimenter,  and  (2)  a  stimulus  display  CRT 
which  is  viewed  by  the  experimental  subject. 

The  subject  is  seated  in  front  of  the  stimulus  CRT,  with  head- 
holder  and  chin  rest,  and  with  the  distance  between  the  eye  and  the  CRT 
surface  adjusted  so  that  12  inches  in  either  the  vertical  or  horizontal 
dimension  on  the  scope  corresponds  to  20°  of  visual  angle.  (An  alternative 
to  the  head-holder/chin  rest  is  to  use  the  Systems  Technology  eye  point- 
of-regard  system  that  essentailly  monitors  the  subject's  head  position 
through  a  movable  attachment  to  a  bite-piece  held  in  the  subject’s  mouth. 
The  combined  inputs  from  the  head-position  monitor  and  the  Biometrics 
eye-position  monitor  are  then  translated  into  X  and  Y  outputs  indicating 
the  eye  point-of-regard . )  The  subject  is  asked  to  fixate  on  a  spot 
displayed  on  the  center  of  the  CRT.  Offset  voltages  are  adjusted  in 
the  X  and  Y  lines  so  as  to  center  the  monitored  eye  location  on  the 
stimulus  spot  on  the  experimenter's  CRT.  The  subject  is  then  asked 
to  fixate  another  spot  located  10  (or  6  inches)  to  the  nasal  side  of  the 
central  fixation  point.  The  X  gain  is  then  adjusted  to  make  the  stimulus 
spot  and  the  eye-fixation  monitor  agree.  The  same  procedure  is  followed 
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with  stimulus  spots  located  6  inches  above,  6  inches  to  the  temporal 
side  and  6  inches  below  the  central  fixation  spot.  Insofar  as  discrepancies 
appear  between  the  amplifier  gains  suitable  for  the  nasal  6  inches  and 
the  temporal  six  inches,  on  the  one  hand,  or  between  the  upper  6  inches 
and  the  lower  six  inches,  on  the  other  hand,  the  issue  is  resolved  in 
favor  of  accuracy  in  the  nasal  sector  in  the  case  of  horizontal  measure¬ 
ments,  and  in  favor  of  accuracy  in  the  lower  sector  in  the  case  of  the 
vertical  measurements.  Having  run  through  these  adjustments  several 
times,  we  then  either  choose  to  accept  the  remaining  nonlinearities 
or  we  may  elect  to  attempt  differential  linearization  of  subdivisions 
of  the  visual  space. 

Now,  the  subject  is  presented  with  a  cartoon-type  stimulus  (out  of 
digital  storage)  on  his  display  scope  and  instructed  to  examine  it  freely 
for  a  specified  number  of  seconds,  after  which  the  cartoon  disappears 
and  the  central  fixation  spot  is  restored.  While  the  subject  is  exam  ring 
the  cartoon,  the  X  and  Y  voltages  from  the  eye  monitor  are  entered  into 
X  and  Y  voltage  distribution  tables  (see  Figure  V.l).  The  entries  in 
the  tables  are  cumulated  in  accordance  with  a  sliding  time-window  (the 
width  of  the  window  being  specified  in  milliseconds  by  the  experimenter). 

The  peak  of  each  distribution  (X  and  Y)  is  treated  as  the  best  estimate 
of  the  eye  pointing  during  the  time  constant  characterizing  the  width 
of  the  sliding  window.  This  X,Y  location  is  made  by  the  center  of  a 
circle  (solid  line)  the  radius  of  whicli  is  controllable  by  the  experimenter. 
We  sometimes  refer  to  this  circle  as  a  "fixation  circle"  or  as  a  "target 
circle"  depending  upon  the  experimental  use  being  made  of  it.  1 f  we 
set  minimum  amplitude  criteria  for  the  distribution  peaks,  we  have  a 
basis  for  monitoring  the  "dwell  time"  or  "spatial  invariance"  as  a  function 
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Figure  V.l  Real-time  eye- f i xa t inn  monitor. 

CRT  display  oi  \  and  Y  d  ist  ribut  ions, 
instantaneous  eye  position  (small  square), 
target  circle  (solid  perimeter),  variance- 
circle  (dotted  perimeter),  prior  lixation 
(cross),  and  OX  indicating  that  the  square 
has  been  inside  the-  solid  circle  leir  the- 
required  number  oi'  milliseconds. 

of  time,  this  is  a  reasonably  effective-  way  e > i  eh  lining  the  beginning 
of  each  fixation.  The  method  for  defining  the  e-nel  o I  a  fixation  is 
described  in  the  next  paragraph. 

A  second  circle,  sharing  the-  same-  center  .is  the  targe  t  circle, 
has  an  independently  variable  radius  which  is  espial  te>  eir  greater 
than  that  of  the  target  circle.  This  e  ire-le-,  which  appears  e>n  the- 
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monitor  scope  as  a  dotted  line  (see  Figure  V.l),  is  used  to  set  a 
spatial  variance  limit  around  the  target  circle.  When  the  instantaneous 
eye-position  exceeds  the  perimeter  of  the  variance  circle,  it  is  assumed 
that  the  eye  is  making  a  saccadic  movement,  or  at  least,  that  it  is 
moving  sufficiently  far  away  that  the  information  accumulated  in  the 
distributions  is  irrelevant,  so  the  distributions  are  re-set  to  zero 
and  the  existing  target  circle  vanishes. 

Another  important  feature  of  this  program  is  the  arrangement 
whereby  if  the  instantaneous  eye  position  (the  small  square  inside  the 
target  circle  in  Figure  V.l)  remains  inside  the  target  circle  for  a 
specified  number  of  milliseconds  (determined  by  the  experimenter),  a 
voltage  appears  on  an  appropriate  analog  output  line  and  the  word  ON 
is  displayed  on  the  monitor  scope  until  the  instantaneous  eye  position 
travels  outside  the  target  circle;  (hen  the  output  voltage  goes  to 
ground  the  the  word  OFF  replaces  the  word  ON  (see  Figure  V.2). 

This  program  provides  for  the  storage  of  100  fixation  locations 
or  100  specified  target  centers.  In  the  analysis  of  a  scanpath,  a 
cross  appears  at  each  fixation  local  ion  and  each  cross  is  numbered 
in  accordance  with  the  ordinal  position  of  the  fixation  in  the  scanpath. 

We  are  also  studying  whether  guiding  the  subject  to  his  earlier 
scanpath  fixation  locations  has  a  favorable  effe?t  upon  image  recognition. 
We  are  entertaining  thT  hypothesis  that  one  feature  of  superior  concrete 
memory  is  an  unusual  consistency  in  retracing  sc.mpaths .  In  other  words, 
a  possible  difference  between  good  concrete  memory  and  poor  concrete 
memory  is  that  the  former  remembers  not  only  what  was  seen  but  also 
where  it  was  seen,  whereas  thfe  latter  may  remember  what  but  not  where. 

II  this  were  true,  guiding  the  latei  to  the  appropriate  location  should 


Figure  V.2  Kc 1  —  t  inn-  lix.il  ion  monitor. 


Similar  to  Figure  V.l  except  that  the 
instantaneous  eye  position  is  outside 
the  target  circle.  OKI-  means  "oil 
target"  and  indicates  that  the  associated 
analog  output  voltagi  is  at  ground. 

enhance  recognition  of  the  contents  ol  scenes  previously  examined 

(scanned) . 
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VI.  MODELING  OF  EEG  SIGNALS  DURING  PHOTIC  STIMULATION 


mm 


1 .  Introduction 

To  better  understand  the  relation  between  visual  stimuli  and  the 
EEG  signal,  we  have  used  a  nonlinear  oscillator  to  model  the  stimulus- 
response  relationship.  In  particular,  we  are  studying  the  entrainment 
of  the  alpha  rhythm  by  periodic  photic  stimuli.  This  will  enhance  our 
understanding  of  the  effect  on  EEG  signals  exerted  by  the  repetitive 
photic  stimulation  and  the  various  degrees  of  efficiency  of  the  stimulus 
falling  on  various  phases  of  the  EEG  signal.  There  is  every  reason  to 
believe,  based  on  the  results  of  other  researchers,  that  the  eye  takes 
in  information  only  when  it  is  fixating.  One  of  our  goals  is  to  guide 
the  eye  to  fixate  only  at  those  instances  corresponding  to  the  EEG 
phases  while  the  brain  is  in  the  best  state  for  processing  information. 
We  believe  that  there  are  relations  between  the  effc'-  • ve  phases  for 
stimulation  and  the  optimal  phases  for  fixations.  It  is  hoped  that  the 
development  of  a  model  for  the  entrainment  process  will  elucidate  the 
the  behavior  of  the  alpha  frequency  and  phase,  and  thus  lead  to  a  more 
practical  brain  state  estimator  and  predictor  for  use  in  our  memory 
studies. 

During  the  past  year,  significant  progress  was  made  toard 
establishing  such  a  model  for  a  class  of  EEG  phenomena.  The  work  began 
with  a  search  for  an  alertness  estimator  for  use  in  interpreting  EEG 
aata  taken  during  eye-mcvement  studies,  since  one  of  the  major  factors 
influencing  the  data  collected  is  drowsiness  or  level  of  alertness 
of  the  subject.  It  was  thought  that  the  degree  of  alpha  rhythm  entrain¬ 
ment  by  periodic  photic  stimuli  might  be  a  good  alertness  index.  A 
nonlinear  model  of  the  entrairment  phenomenon  was  developed,  and  has 
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subsequently  predicted  a  number  of  unexpected  related  phenomena.  We 
will  discuss  the  model,  the  EEG  data,  and  simulations  of  the  model. 


2.  A  Mathematical  Model  for  Photic  Entrainment  of  EEG  Signals 

We  will  show  that  the  entrainment  behavior  of  the  alpha  rhythm  can 
be  modeled  by  a  type  of  nonlinear  oscillator  first  studied  by  Van  der 
Pol.  The  model  is  concerned  primarily  with  the  nonrandom  periodic 
component  of  the  EEG,  although  it  is  hoped  that  some  light  will  be 
shed  on  transitionary  behavior,  such  as  visual  evoked  responses. 

We  denote  the  EEG  signal  by  u(t)  which  is  assumed  to  -.atisfy 
the  nonlinear  differential  equation  below  during  periodic  stimulation: 


u  -  e(l  - 


2,  • 

u  )  u 


,  2 
+  w  u 


-  B  cos  vt  1  B 


0 


(1) 


where  to  is  the  unstimulated  alpha  frequency,  v  is  the  stimulus 
frequency,  B  is  the  stimulus  amplitude,  B^  is  the  stimulus  bias  level, 
and  e  controls  the  amount  of  nonlinearity  in  the  system.  It  is  proposed 
that  e  is  related  to  a  certain  psychophysiologic  state  and  mathematically 


represents  the  amount  of  coupling  between  the  brain  and  its  environment. 
This  equation  assuaes  a  sinusoidal  stimulus  (typically  B^  =  B)  since 
the  analysis  is  simple.  Our  experiments  to  date  have  used  a  pulsed 
stimulus,  and  the  resulting  differences  will  be  discussed  in  the 
section  on  simulation. 


Letting  x  =  to  u  -  Bq  in  ( 1 ) ,  we  get 
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where 
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,  (3  = 
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For  simple  entrainment,  where  v  ,  we  expect  a  solution  of  the 


xQ(t)  =  sin  vt  +  cos  vt 


Substituting  (3)  into  (2): 


2  2 
-b^v  sin  vt  -  b^v  cos  vt 

-u(l  -  (3b^  sin  vt  -  (3b2  cos  vt 

2  2  2  2 
“7b1  sin  vt  -  2>bib^  sin  -t  cos  vt  -  yb^  cos  vt)  .< 

(b^v  cos  vt  -  b^v  sin  vt) 

2  2  2 
+  tu  b^  sin  vt  +  uj  b2  cos  vt  -  uj  B  cos  vt  =  0 

With  a  little  trigonometry,  we  can  write  this  in  a  form  useful  for 
harmonic  balancing: 


^7b2V  +  ^7bib2v  +  °j2b] 


+  sin  2vt 
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+  cos  vt 
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+  cos  2vt 


+  cos  3vt 


=  0  . 


As  a  first  approximation,  wo  force  the  sin  vt  and  cos  vt  terms 
to  zero  (harmonic  balancing) : 


bl(w  -  v2)  +  nb2v  -  ~  H7b2b2v  -  •  ixyb^v  =  0 

b2<w  -  v  )  -  pb^  v  -)  -i-  p7  b^b^v  +  P7b^v  =  o2B 


or 
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Eliminating  x^.y^,  we  get 


,  2  2 
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(5) 


Equation  (5)  generates  the  response  curves  for  the  harmonic  oscillations 

which  are  plots  of  the  amplitude  as  a  function  of  the  detuning  a 

for  various  values  of  the  stimulus  parameter  B.  When  B  =  0  ,  we  have 

2  2  2 

ri  =  i.  01  =  0,  and  bx  +  b2  =  a()  ,  hence  a^  is  the  amplitude  of  the 

autonomous  oscillation  in  (3). 

There  are  two  solutions  when  the  detuning  o  =  0;  we  have  seen 
the  trivial  case  above,  and  he  other  case  occurs  when  y  =0. 
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When  this  happens,  x  is  the  solution  to 
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_uj  B 
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Here  the  oscillation  is  xq  =  sin  vt  ;  hence,  them  is  a  90° 
phase  difference  between  the  stimulus  and  EEC  signals  during  driving 
at  the  alpha  frequency.  As  we  shall  see  later,  this  has  been  borne 
out  by  the  analysis  of  real  data. 

lor  the  case  of  subharmonic  and  superharmonic  entrainment,  the 
solution  is  of  the  form  xQ  =  bQ  cos  wt  +  ^  sin  nvt  +  b2  cos  nvt, 

where  n  is  l/2,  1/3 . or  2,  3,  ...,  respectively.  The  analysis 

is  quite  similar  to  the  one  above. 

We  shall  next  describe  our  simulation  of  this  model  and  the 
resultant  phenomena  it  produces.  The  simulated  model  predicts  and 
accounts  for  many  of  the  phenomena  exhibited  by  real  data  presented 
in  the  last  section. 

3-  Simulation  of  EEG  g  Photic  Driving 

The  simulation  is  a  digital  computer  implementation  of  the  Van 
dor  Pol  oscillator  described  above.  A  fourth  order  Runge-Kutta  formula 
is  used  to  solve  the  differential  equation,  and  an  FIT  is  used  to  compute 
the  spectrum  of  the  solution.  In  the  figures  to  follow,  the  upper  plot 
shows  the  simulated  EEG  signal  at  the  time  the  stimulus  was  initiated, 
Plus  the  stimulus  waveform  itself.  The  lower  plot  contains  the  power 
spectra  of  the  EEG  and  stimulus  waveforms.  All  simulations  were  done 
with  identical  parameter  settings. 

The  stimulus  used  in  the  simulations  is  a  pulse  train  since  our 
experimental  stimulus  has  been  a  stroboscope.  The  effect  of  the  pulsed 
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stimulus  is  akin  to  that  from  a  set  of  sinusoidal  stimuli  placed  at 
integer  harmonics.  Note  that  superposition  does  not  apply  here. 

Figure  VI . 1  shows  the  autonomous  oscillation  and  its  spectrum. 

The  stimulus  is  totally  absent  here;  the  amplitude  of  the  oscillation 
is  fixed  by  the  limit  cycle  phenomenon.  This  situation  obviously 
corresponds  to  the  closed-eye  alert  brain  state.  Figure  VI. 2  illustrates 
entrainment  to  a  9.0  Hz.  stimulus;  the  alpha  frequency  is  9.7  Hz.  Note 
the  small  response  at  the  alpha  frequency.  In  addition,  there  is  a 
very  small  response  at  18.0  Hz.,  but  apparently  the  simulation  parameters 
do  not  match  the  subject  perfectly  since  its  amplitude  should  be  larger 
(see  Figure  VI. 7).  Parameter  matching  is  difficult  since  the  alertness 
of  the  subject  varies  widely  and  has  a  strong  influence  on  the  resulting 
spectrum.  Figure  VI. 3  is  a  combined  frequency  oscillation  which  correctly 
predicts  a  phenomenon  not  previously  reported  in  EEG  literature  (see 
Figure  VI. 8).  Again,  the  second  harmonic  amplitude  is  somewhat  low. 

Figure  VI. 4  predicts  Figure  VI . 9  remarkably  well;  this  is  another  com¬ 
bined  frequency  oscillation.  In  Figure  VI. 5,  we  see  a  superharmonic 
entrainment  (some  of  this  is  due  to  the  pulsed  stimulus).  Note  the 
responses  at  4.5,  9.0,  13.5,  and  9.7  Hz.,  and  compare  with  Figure  VI. 10. 
The  simulation  of  subharmonic  entrainments  is  in  progress. 

4 .  EEG  During  Photic  Driving 

The  EEG  potentials  are  measured  between  the  occipital  region  and 
the  yoked  left  and  right  earlobes  with  the  ground  electrode  over  the 
mastoid  process.  Left  and  right  occipital  potentials  are  recorded 
on  analog  tape  in  addition  to  the  analysis  procedures  to  be  described. 

The  subject  is  stimulated  through  closed  eyelids  with  a  stroboscope 
for  30  seconds  on,  then  30  seconds  off.  The  frequency  is  varied  between 
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.5  and  20  Hz.  in  Severn’  patterns.  In  addition,  the  subject  is  requested 
to  press  a  key  once  every  five  seconds.  The  variance  of  the  inter¬ 
key-press  time  provide  a  primitive  alterness  index. 

The  LEG  data  is  lowpass  filtered  with  a  cut-off  frequency  of  30 
Hz.,  and  its  power  spectrum  is  computed  by  a  SAICOR  digital  spectrum 
analyzer  and  written  on  9-track  magnetic  tape  by  a  PDP-8.  The  spectra 
shown  below  are  cumulative  over  the  stimulus-on  period. 

Figure  VI. 6  is  a  60-second  cumulative  spectral  of  the  subject's 
unstimulated  alert  EEG.  The  alpha  frequency  here  is  estimated  at  9.7 
Hz.  In  Figure  VI. 7  we  see  the  result  of  a  stroboscopic  stimulus  at 
9.0  Hz.  Note  that  the  alpha  rhythm  has  entrained  to  9.0  Hz.,  and  that 
a  harmonic  at  18.0  Hz.  also  appears.  In  Figure  VI. 8  about  half  the  power 
lies  at  the  alpha  frequency,  and  the  rest  Is  at  the  stimulus  frequency 
of  8.5  Hz.  and  its  second  harmonic  ef  17.0  Hz.  We  can  say  that  the  edge 
of  entrainment  here  is  about  9  Hz.,  and  that  we  are  now  seeing  a  combined 
frequency  oscillation  where  the  stimulus  is  at  11.0  Hz.  Figure  VI . 10 
shows  a  superharmonic  entrainment  to  a  stimulus  of  4 . 5  Hz.  Much  of  this 
effect  is  due  to  the  integer  harmonics  of  the  pulsed  stimulus,  as 
discussed  in  the  simulation  section.  Note  the  small  response  at  4.5 
and  13.5  Hz.  In  Figure  VI. 11,  we  see  a  subharmouic  entrainment  due  to 
stimulation  at  18.0  Hz.  Note  the  small  combined  frequency  effect. 

The  spectral  data  is  well  predicted  by  the  nonlinear  model; 
harmonic  entrainment,  combined  frequency  responses,  superharmonic 
entrainment,  and  subharmonic  harmonic  entrainment  are  all  accounted  for. 

We  now  turn  to  the  90°  phase  prediction. 

For  phase  calculations  the  EEG  data  is  bandpass  filtered  with  a 
bandwidth  of  5  Hz.  and  a  center  frequency  at  the  subject's  mean  alpha 
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frequency.  The  parameters  of  interest  are  computed  online  and  stored 
digitally  on  magnetic  tape  in  addition  to  being  output  on  a  strip  chart 
recorded  with  the  EEG. 

The  mean  alpha  period  is  determined  from  the  period  oi  its  auto¬ 
correlation  function,  which  is  obtained  once  every  160  milliseconds 
from  a  digital  correlation  analyzer.  The  internal  integration  step  is 
expoi  entially  weighted  with  a  time  constant  of  4  seconds  to  provide  a 
reasonably  stationary  period.  The  phase  between  the  alpha  rhythm  and 
the  periodic  stimulus  is  obtained  by  cross  correlation,  again  with  a 
time  constant  of  4  seconds.  The  hardware  correlational  analyses 
provides  a  simple  real-time  evaluation  of  two  key  parameters  and 
minimize  the  computation  performed  by  the  computer  running  the  expeiiment. 

We  show  some  results  of  this  analysis  in  Figure  VI. 12.  Note  the 
changes  in  the  alpha  period  and  phase  lag  as  a  function  of  the  stimulus 
being  on  or  off.  When  the  stimulus  is  on,  the  alpha  period  becomes 
100  milliseconds,  the  period  of  the  stroboscopic  stimulus.  When  the 
stimulus  is  off,  the  period  becomes  88  milliseconds,  the  autonomous 
alpha  period  of  the  subject.  In  addition  the  phase  between  the  stimulus 
and  the  alpha  rhythm  assumes  a  fairly  stationary  value  of  90  degrees 
when  the  stimulus  is  on  and  drifts  at  no  particular  value  when  the 
stimulus  is  off. 

These  experimental  data  clearly  exhibit  three  of  the  major 
characteristics  as  predicted  by  the  nonlinear  oscillator  model:  the 
autonomous  alpha  frequency  w  is  entrained  to  the  stimulus  frequency 
v;  the  phase  lag  assumes  a  preferred  value  of  90  degrees;  and  the 
autonomous  rhythm  to  •'eturns  when  the  stimulus  is  removed. 


55 


VII.  PHASE  CONTINGENT  AVERAGES 


Evoked  response  averaging  is  a  widely  used  method  for  extracting 
stimulus-coherent  signals  from  random  noise.  As  we  suspected  the 
traditional  method  for  averaging  conceals  other  interesting  features 
of  the  VER  which  can  be  preserved  through  the  application  of  a  new 
method  of  averaging  called  phase  contingent  expansion. 

The  coincidence  between  a  photic  flash  stimulus  and  EEG  alpha 
phase  can  be  expected  to  be  statistically  random  if  the  flashes  are 
delivered  at  arbitrary  intervals  and  if  the  minimum  interval  between 
a  single  flash  or  group  of  flashes  is  sufficiently  long  to  allow  for 
the  complete  dissipation  of  the  effects  of  the  previous  flash  or  flashes. 
Thus,  for  the  traditional  visual  evoked  response,  the  component  sample 
epochs  are  initiated  with  the  flash  falling  with  equal  probability 
(equal  frequency)  on  all  possible  phases  of  alpha.  In  other  words,  the 
traditional  AEP  is  an  average  conditional  only  upon  the  occurrence  of 
the  flash,  and  assumes  that  the  response  looked  for  in  the  averaging 
process  is  stimulus  locked.  However,  if  the  cortical  response-  '•o  the 
flash  were  also  influenced  by  the  phase  of  the  alpha  rhythm  at  the  time 
of  the  flash,  some  modification  of  the  averaging  technique  would  be 
required  to  bring  out  the  phase  contingent  properties  of  the  VER. 

The  phase  contingent  expansion  of  the  visual  evoked  response 
described  here  is  such  a  method.  By  regrouping  sample  epochs  of  the 
VER  according  to  the  phase  of  the  alpha  rhythm  at  the  time  of  each 
flash  stimulus,  a  complete  series  of  phase  contingent  sub— averages 
contairing  all  component  data  sample  epochs  in  the  VER  is  obtained. 

The  sub-averages  are  displayed  according  to  selected  flash-phase  coinci¬ 
dence  ranges  in  ascending  phase  order. 
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Quadrature  analysis  of  tne  EEG  alpha  rhythm  is  accomplished 

using  a  somewhat  simplified  version  of  a  technique  described  in  a  previous 

report.  The  EEG  signals  are  first  band  limited  (by  the  transversal 

filter  mentioned  in  Data  Analysis)  to  include  the  subject's  alpha 

rhythm.  Reference  cosine  and  sine  waves  using  the  subject's  mean  alpha 

frequency  are  generated  with  the  condition  that  for  each  sample  epoch 

of  EEG  data,  the  peak  of  the  cosine  wave  is  always  coincident  with 

the  stimulus  (or  trigger)  of  the  sample  epoch.  Thus,  the  cosine  and 

sine  waves  are  stimulus-locked  coherent  rhythms  which,  by  the  quadrature 

technique,  will  yield  the  continuous  phase  relationship,  $  (•),  of 

Q 

these  waves  to  the  sample  epoch  in  question.  The  quadrature  approach  to 
alpha  phase  classification  is  summarized  in  Figure  VI 1 . 1 . 

Ai  example  of  a  phase  contingent  expansion  of  a  visual  average 
evoked  potential  is  shown  in  Figure  VII. 2.  In  this  example,  the  sample 
epochs  of  EEG  data  were  classified  into  twenty  phase  ranges  covering 
°n *  complete  cylce  of  the  alpha  rhythm.  The  sample  epochs  in  each  phase 
range  were  then  averaged  and  normalized.  The  phase  contingent  averages 
are  arranged  in  ascending  order  beginning  with  the  surface  negative 
peak  and  ending  with  the  next  surface  negative  peak  of  the  alpha  cycle. 

It  can  be  seen  that  there  is  .  systematic  adjustment  in  each  of  the 
contingent  averages  to  achieve  a  position  deflection  in  the  region  of  the 
dominant  P  wave  in  the  grand  mean  (displayed  as  a  heavier  function  at 
the  top  of  the  phase  contingent  series).  By  comparison  of  the  above 
series  with  a  similar  phase  contingent  series  (Figure  VII. 3)  obtained 
in  the  absence  of  photic  stimulation  (there  was  a  trigger  pulse  for  the 
purpose  of  phase  classification  but  the  strobe  was  inhibited),  we 
see  that  the  greatest  phase  shifting  of  the  alpha  rhythm  in  response 
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VII. 2  Phase  contingent  averages  of  visual  evoked  responses. 

I ach  sample  epoch  was  r  Miss  1  lied  with  respect  to  the  phase  ol  the 
alpha  rhythm  at  t  lie  moment  tin  flash  stimulus  was  delivered.  The 
grand  mean  is  the  traditional  average  of  all  the  epochs  without 

regard  to  phast . 


CONTINGENT  AVERAGES 


TRIGGER 


to  photic  stimulation  occurs  in  the  mid-range  of  the  alpha  cycle.  We 
are  currently  studying  the  signal-to-noise  properties  of  the  phase 
contingent  sample  epochs  in  the  time  region  of  the  dominant  P  wave. 

Our  preliminary  results  suggest  that  the  signal-to-noise  ratio  is  the 
poorest  in  the  mid-range  of  the  alpha  cycle  where  the  greatest  shifting 
of  the  autonomous  alpha  phase  must  take  place  in  response  to  photic 
stimulation.  This  agrees  with  the  observation  of  various  investigators 
that  reaction  times  are  longer  and  sensory  thresholds  are  higher  for 
stimuli  delivered  at  these  phase  contingencies  than  for  stimuli  that  are 
contingent  with  surface  negative  peak  phases.  These  studies  will  be 
reported  in  greater  detail  in  a  later  report. 
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VIII.  CONCLUSION 


As  described  in  this  report,  our  work  has  been  proceeding  towards 
the  goals  that  we  proposed.  We  now  have  better  techniques  to  deal  with 
the  errors  and  distortions  of  the  eye-movement  data  as  measured  by  the 
Biometrics  Eye-Movement  Monitor  unit.  We  believe  that  we  have  utilized 
the  Biometrics  unit  to  the  best  of  its  capability.  Some  prediction  schemes 
which  predict  the  eye-fixation  points  have  been  implemented  and  new  ones 
are  being  developed.  A  large  amount  of  eye-movement  data  has  been 
collected  and  is  being  analyzed.  The  analysis  will  give  us  a  better 
understanding  of  the  characteristics  of  the  saccadic  movement  and  the 
fixation.  It  is  expected  to  lead  to  better  practical  predictive  models. 

A  new  experiment  paradigm  has  been  set  up  for  the  study  of  visual 
scanpaths  associated  with  both  superior  and  inferior  imagery.  A  real¬ 
time  scanpath  analyzer  has  been  developed  and  implemented.  In  the  offing, 
we  shall  see  new  interesting  results  in  the  scanpath  study  with  the  aim 
to  find  the  scanpath  which  will  result  in  superior  memory.  We  have 
found  a  definite  excitability  cycle  correlated  with  the  phase  of  the 
alpha  rhythm  at  one  instant  of  photic  stimulation  via  a  phase  contingent 
analysis  of  visual  evoked  responses.  An  EEC  signal  model  has  been  con¬ 
ceived  and  studied  with  the  hope  that  it  will  aid  us  in  building  better 
tracking  models  for  the  brain  states.  We  have  also,  in  this  period, 
expanded  our  computer  system  to  better  suit  our  needs,  and  completed  a 
computer  program  dubbed  ’  .SEEK"  that  was  designed  to  open  up  a  new 
avenue  for  the  study  of  visual  scanpaths. 

With  the  continuously  up-dated  information  concerning  the  eye 
position  and  brain  state  for  adjusting  the  stimulus  parameters  and  the 
tracking  and  prediction  techniques,  we  should  be  able  to  guide  the  eyes 


to  fixate  at  the  optimal  locations  and  at  the  optimal  time  instants  such 
that  the  vividness  and  persistence  of  the  desired  after-images  will  be 
enhanced.  Future  works  are  planned  along  the  lines  to  get  a  better 
handle  of  these  optimal  locations  and  optimal  time  instants.  Armed  with 
this  information,  we  shall  attempt  to  devise  real-time  strategies  for 
the  control  of  visual  target  impression  and  visual  image  (memory) 
persistence  and/or  recall  ability  to  complete  our  design  of  the  closeiy- 
coupled  man-machine  system  for  visual  memory  tracking  and  training. 
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